Methods of interfacing with multi-input devices and multi-input display systems employing interfacing techniques

ABSTRACT

Methods and systems for interfacing with multi-input devices employ various techniques for controlling the window framing of images. Such techniques provide control, including moving, sizing, and orientating, of one or more displayed window frames in which one or more images are displayed.

REFERENCE TO RELATED APPLICATION

This application claims priority to U.S. Provisional Patent ApplicationNo. 61/043,863, filed Apr. 10, 2008, the disclosure of which isincorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to methods of interfacing with multi-inputdevices, including multi-touch input devices, and multi-input displaysystems employing interfacing techniques.

2. Description of the Related Art

So-called touch screen displays are well known and common in manyindustries. As an example, most bank ATMs use touch screen displays. Asthe utility of these types of displays increases, displays within homeand business, including televisions and computer monitors, areincreasingly becoming touch sensitive.

Recently, multi-touch input devices are becoming more available forvarious uses due to advancements in touch sensing technologies,reductions in cost, and other factors. Multi-touch input devices, bydefinition, are capable of detecting multiple inputs simultaneously.U.S. patent application Ser. Nos. 11/833,908; 12/185,782; 12/182,440;60/821,325; 60/953,966; and 60/952,709; and U.S. Patent Publication Nos.2008/0029691A1; 2008/0284925A1; and 2009/0033637A1, all assigned to theassignee of the present application and incorporated herein byreference, identify various technologies and publications within thefield, and further describe additional advancements in multi-touchsensing through frustrated total internal reflection.

Interface control technology, including interfacing techniques, has beenwidely studied. The following publications explore various concepts,methodologies and techniques in this area: Buxton, W., Hill, R., andRowley, P., “Issues and Techniques in Touch-Sensitive Tablet Input,”Proceedings of the 12th Annual Conference on Computer Graphics andInteractive Techniques SIGGRAPH '85, ACM Press, New York, N.Y., 215-224(1985); Michael Chen, “A Study in Interactive 3-D Rotation Using 2-DControl Devices,” Computer Graphics, Vol. 22, No. 4, 121-129 (1988); K.Shoemake, “ARCBALL: A User Interface for Specifying Three-DimensionalOrientation Using a Mouse,” Proceedings of the conference on Graphicsinterface '92, 151-156 (1992); Ken Hinckley, “Haptic Issues for VirtualManipulation,” A Dissertation Presented to the Faculty of the School ofEngineering and Applied Science at the University of Virginia, section6.1-6.3 (1996), from the websitehttp://research.microsoft.com/Users/kenh/thesis/front.htm; Dietz, P. andLeigh, D., “DiamondTouch: A Multi-User Touch Technology,” Proceedings ofthe 14th Annual ACM Symposium on User Interface Software and Technology(Orlando, Fla., Nov. 11-14, 2001), UIST '01. ACM Press, New York, N.Y.,219-226 (2001); Lee, S., Buxton, W. and Smith, K. C., “A Multi-TouchThree Dimensional Touch-Sensitive Tablet,” Proceedings of the SIGCHIConference on Human Factors in Computing Systems (San Francisco, Calif.,United States), CHI '85. ACM Press, New York, N.Y., 21-25 (1985); Malik,S. and Laszlo, J., “Visual Touchpad: A Two-Handed Gestural InputDevice,” Proceedings of the 6th International Conference on MultimodalInterfaces (State College, Pa., USA, Oct. 13-15, 2004), ICMI '04. ACMPress, New York, N.Y., 289-296 (2004); Rekimoto, J., “SmartSkin: AnInfrastructure for Freehand Manipulation on Interactive Surfaces,”Proceedings of the SIGCHI Conference on Human Factors in ComputingSystems, CHI '02, ACM Press, New York, N.Y., 113-120 (2002); Westerman,W., Elias, J. G., and Hedge, A., “Multi-Touch: A New Tactile 2-D GestureInterface for Human-Computer Interaction,” Proceedings of the HumanFactors and Ergonomics Society 45th Annual Meeting (Minneapolis/St.Paul, Minn., October 2001), 632-636 (2001); Wilson, A. D., “TouchLight:An Imaging Touch Screen and Display for Gesture-Based Interaction,”Proceedings of the 6th International Conference on Multimodal Interfaces(State College, Pa., USA, Oct. 13-15, 2004), ICMI '04. ACM Press, NewYork, N.Y., 69-76 (2004); and Wu, M. and Balakrishnan, R., “Multi-Fingerand Whole Hand Gestural Interaction Techniques for Multi-User TabletopDisplays,” Proceedings of the 16th Annual ACM Symposium on UserInterface Software and Technology (Vancouver, Canada, Nov. 2-05, 2003),UIST '03, ACM Press, New York, N.Y., 193-202 (2003), each of which isincorporated herein by reference.

Various publications explore two-handed input. These include: R.Balakrishnan and K. Hinckley, “Symmetric bimanual interaction,” CHI '00:Proceedings of the SIGCHI conference on Human factors in computingsystems, 33-40 (2000); R. Balakrishnan and G. Kurtenbach, “Exploringbimanual camera control and object manipulation in 3D graphicsinterfaces,” CHI '99: Proceedings of the SIGCHI conference on Humanfactors in computing systems, 56-63 (1999); Y. Guiard, “Asymmetricdivision of labor in human skilled bimanual action: The kinetic chain asa model,” Journal of Motor Behavior, 19(4):486-517 (1987); K. Hinckley,R. Pausch, J. C. Goble, and N. F. Kassell, “Passive real-world interfaceprops for neurosurgical visualization,” CHI '94: Proceedings of theSIGCHI conference on Human factors in computing systems, 452-458 (1994);G. Kurtenbach, G. Fitzmaurice, T. Baudel, and B. Buxton, “The design ofa GUI paradigm based on Tablets, Two-hands, and Transparency,” CHI '97:Proceedings of the SIGCHI conference on Human factors in computingsystems, 35-42 (1997); I. Llamas, B. Kim, J. Gargus, J. Rossignac, andC. D. Shaw, “Twister: a space-warp operator for the two-handed editingof 3D shapes,” ACM Transactions on Graphics, 22(3):66 3-668 (2003); andR. Zeleznik, A. Forsberg, and P. Strauss, “Two pointer input for 3Dinteraction,” SI3D '97: Proceedings of the 1997 symposium on Interactive3D graphics, 115-120 (1997). Each of these publications is incorporatedherein by reference.

Recent patent publications in this field further include U.S. PatentPublication Nos. 2006/0026521, 2006/0026535, 2006/0053387, 2006/0085757,2006/0033724, 2006/0161870, 2006/0161871, and 2006/0026535, each ofwhich is incorporated herein by reference. Moreover, U.S. PatentPublication Nos. 2008/0180404A1, 2008/0180405A1 and 2008/0180406A1,assigned to the assignee of the present application and incorporatedfully herein by reference, describe particular advancements in both 2Dand 3D object control and 3D globe view control, among other things.

SUMMARY OF THE INVENTION

Human interface techniques that take full advantage of the multi-inputcapability of multi-input sensing technology need to be developed. It istherefore a general object of the present invention to provide methodsand systems for facilitating human interfacing with multi-inputtechnology, such as multi-touch tablets and multi-touch display devices.More specific objects of the present invention are discussed in thedetailed description section provided below.

In accordance with the present invention, a method of interfacing with amulti-input display device comprises displaying on a multi-input displaydevice a graphical window and at least a first portion of an imagewithin the graphical window, identifying a number, position and movementof one or more elements contacting the display device, changing acharacteristic of the displayed graphical window in accordance with theidentified number, position and movement of the one or more elementscontacting the display device, the characteristic corresponding to afirst characteristic when only one element is identified as contactingthe display device, the first characteristic being a position of thedisplayed graphical window, and the characteristic corresponding to asecond characteristic when a plurality of elements are identified ascontacting the display device, the second characteristic being at leasta shape and/or size of the displayed graphical window. The first andsecond characteristics are different. The method further comprisesdisplaying on the display device at least a second portion of the imagewithin the graphical window having the changed characteristic, andpositions on the display device of common portions of the first andsecond portions of the image are substantially the same.

In accordance with another embodiment of the present invention, a methodof interfacing with a multi-input display device comprises displaying ona multi-input display device a graphical window and at least a firstportion of an image within the graphical window, identifying a number,position and movement of a plurality of elements simultaneouslycontacting the display device, manipulating the displayed graphicalwindow in accordance with one or more modifications, the one or moremodifications comprising a first modification implemented when only twoelements are identified as contacting the display device at positions ona frame of the displayed graphical window, the first modificationchanging at least a shape and/or size of the displayed graphical windowin accordance with the identified positions and movement of the twoelements, a second modification implemented when three elements areidentified as contacting the display device at positions on the frame ofthe displayed graphical window, the second modification changing atleast the position, shape and/or size of the displayed graphical windowin accordance with the identified positions and movement of the threeelements, the first and second modifications being different, displayingon the display device at least a second portion of the image within themanipulated graphical window, and positions on the display device ofcommon portions of the first and second portions of the image beingsubstantially the same.

In accordance with a system embodiment of the present invention, amulti-input display system comprises a display device for displaying ona display surface a graphical window and at least a first portion of animage within the graphical window, the display device adapted to detectone or more elements contacting the display surface, a controller foridentifying a number, position and movement of the one or more elementscontacting the display surface, and for changing a characteristic of thedisplayed graphical window in accordance with the identified number,position and movement of the one or more elements contacting the displaysurface, the characteristic corresponding to a first characteristic whenonly one element is identified as contacting the display surface, thefirst characteristic being a position of the displayed graphical window,and the characteristic corresponding to a second characteristic when aplurality of elements are identified as contacting the display surface,the second characteristic being at least the shape and/or size of thedisplayed graphical window, the first and second characteristics beingdifferent, the controller controlling the display device to display onthe display surface at least a second portion of the image within thegraphical window having the changed characteristic, positions on thedisplay surface of common portions of the first and second portions ofthe image being substantially the same.

In accordance with another system embodiment of the present invention, amulti-input display system comprises a display device for displaying ona display surface a graphical window and at least a first portion of animage within the graphical window, the display device adapted to detecta plurality of elements simultaneously contacting the display surface, acontroller for identifying a number, position and movement of theplurality of elements simultaneously contacting the display surface, andfor manipulating the displayed graphical window in accordance with oneor more modifications, the one or more modifications comprising a firstmodification implemented when only two elements are identified ascontacting the display surface at positions on a frame of the displayedgraphical window, the first modification changing at least the shapeand/or size of the displayed graphical window in accordance with theidentified positions and movement of the two elements, a secondmodification implemented when three elements are identified ascontacting the display surface at positions on the frame of thedisplayed graphical window, the second modification changing at leastthe position, shape and/or size of the displayed graphical window inaccordance with the identified positions and movement of the threeelements, the first and second modifications being different, thecontroller controlling the display device to display on the displaysurface at least a second portion of the image within the manipulatedgraphical window, and positions on the display surface of commonportions of the first and second portions of the image beingsubstantially the same.

For each of the above-summarized embodiments of the present invention,various aspects and features of the present invention may be carriedout. Some of these features and aspects are summarized below.

As an aspect of the invention, when two elements are identified ascontacting the display device initially at positions on the displayedgraphical window, the size of the displayed graphical window isuniformly scaled in accordance with the identified movement of the twoelements.

As a further aspect of the invention, when two elements are identifiedas contacting the display device initially at positions on the displayedgraphical window, the displayed graphical window is moved and sized inaccordance with the identified movement of the two elements.

As a feature of this aspect, the displayed graphical window is rotatedin accordance with changes in orientation of the identified positions ofthe two elements relative to one another.

As an additional aspect of the invention, when three elements areidentified as contacting the display device initially at positions on aframe of the displayed graphical window, the shape of the displayedgraphical window is manipulated based on changes in the identifiedpositions of the three elements in accordance with a best-fitmethodology.

As a feature of this aspect, manipulation of the shape of the displayedgraphical window is a 5-degree of freedom operation.

As yet another aspect of the invention, when first and second elementsare identified as contacting the display device initially at positionson a first edge of a frame of the displayed graphical window and a thirdelement is identified as contacting the display device initially at aposition on a second edge opposite the first edge of the frame of thedisplayed graphical window, the displayed graphical window ismanipulated based on the movement of one or more of the first, secondand third elements.

As yet an additional aspect of the invention, the shape of the displayedgraphical window is modified by moving the first edge of the frame ofthe displayed graphical window in alignment with identified jointmovement of the first and second elements and the second edge of theframe is held in alignment with the identified position of the thirdelement.

As another aspect of the invention, the second edge of the frame of thedisplayed graphical window is moved in alignment with the identifiedmovement of the third element and the first edge of the frame ismaintained at a fixed position.

As a further aspect of the invention, the length of the first and secondedges of the frame of the displayed graphical window are changed inaccordance with a change in distance between the identified positions ofthe first and second elements.

As an additional aspect of the invention, the displayed graphical windowis rotated in accordance with changes in orientation of the identifiedpositions of the first and second elements relative to one another sothat the first edge of the frame of the displayed graphical window isaligned with an axis extending through the first and second elements asthe first and second elements move.

As yet a further aspect of the invention, the displayed graphical windowis rectangular in shape, and manipulating the displayed graphical windowcomprises aligning the first edge of the frame of the displayedgraphical window with an axis extending through the identified positionsof the first and second elements, aligning the first edge (includingremaining stationary, moving vertically, moving horizontally, orrotating), changing a length of the first and second edges of the framein accordance with a change in distance between the identified positionsof the first and second elements, and aligning the second edge of theframe with the identified position of the third element.

In accordance with a further method embodiment of the present invention,a method of interfacing with a multi-input display device comprisesdisplaying on a multi-input display device a graphical window and atleast a first portion of an image within the graphical window,identifying a position and movement of one element contacting an edge ofthe displayed graphical window or a plurality of elements simultaneouslycontacting respectively different edges of the displayed graphicalwindow, manipulating the displayed graphical window in accordance withone or more modifications, the one or more modifications comprising afirst modification implemented when only one element is contacting anedge of the displayed graphical window, the first modification movingthe edge of the displayed graphical window contacted by the element in adirection of identified movement of the one element normal to an axis ofthe edge, a second modification implemented when a plurality of elementsare simultaneously contacting respectively different edges of thedisplayed graphical window, the second modification simultaneouslymoving each of the edges of the displayed graphical window contacted bya respective one of the elements in a direction of identified movementof the respective element normal to an axis of the respective edge,displaying on the display device at least a second portion of the imagewithin the manipulated graphical window, and positions on the displaydevice of common portions of the first and second portions of the imagebeing substantially the same.

In accordance with a further system embodiment of the present invention,a multi-input display system comprises a display device for displayingon a display surface a graphical window and at least a first portion ofan image within the graphical window, the display device adapted todetect one or more elements contacting the display surface, a controllerfor identifying a position and movement of one element contacting anedge of the displayed graphical window or a plurality of elementssimultaneously contacting respectively different edges of the displayedgraphical window, the controller adapted to manipulate the displayedgraphical window in accordance with one or more modifications, the oneor more modifications comprising a first modification implemented whenonly one element is contacting an edge of the graphical window, thefirst modification moving the edge of the graphical window contacted bythe element in a direction of identified movement of the one elementnormal to an axis of the edge, a second modification implemented when aplurality of elements are simultaneously contacting respectivelydifferent edges of the graphical window, the second modificationsimultaneously moving each of the edges of the graphical windowcontacted by a respective one of the elements in a direction ofidentified movement of the respective element normal to an axis of therespective edge, the controller controlling the display device todisplay on the display surface at least a second portion of the imagewithin the manipulated graphical window, positions on the displaysurface of common portions of the first and second portions of the imagebeing substantially the same.

As an aspect of these further method and system embodiments, a positionand movement of three or more elements simultaneously contacting threeor more different edges of the displayed graphical window areidentified, and the second modification simultaneously moves each of thethree or more edges of the displayed graphical window contacted by thethree or more elements in a respective direction of the identifiedmovement of the respective element normal to an axis of the respectiveedge.

In accordance with an additional method embodiment of the presentinvention, a method of interfacing with a multi-input display devicecomprises displaying on a multi-input display device a graphical windowand at least a first portion of an image within the graphical window,identifying positions and movement of a plurality of elementssimultaneously contacting one or more edges of the displayed graphicalwindow, manipulating the displayed graphical window in accordance withone or more modifications, the one or more modifications comprising afirst modification implemented when two of the plurality of elements arecontacting opposite edges of the displayed graphical window, the firstmodification changing at least the position and/or shape of thedisplayed graphical window in accordance with the identified positionsand movement of the two elements contacting the opposite edges, a secondmodification implemented when two of the plurality of elements arecontacting adjacent edges of the displayed graphical window, the secondmodification changing at least the shape and/or size of the displayedgraphical window in accordance with the identified positions andmovement of the two elements contacting the adjacent edges, a thirdmodification implemented when two of the plurality of elements arecontacting a same edge of the displayed graphical window, the thirdmodification changing at least the position and/or shape of thedisplayed graphical window in accordance with the identified positionsand movement of the two elements contacting the same edge, displaying onthe display device at least a second portion of the image within themanipulated graphical window, and positions on the display device ofcommon portions of the first and second portions of the image beingsubstantially the same.

In accordance with an additional system embodiment of the presentinvention, a multi-input display system comprises a display device fordisplaying on a display surface a graphical window and at least a firstportion of an image within the graphical window, the display deviceadapted to detect elements simultaneously contacting the displaysurface, a controller for identifying positions and movement of aplurality of elements simultaneously contacting one or more edges of thedisplayed graphical window, and for manipulating the displayed graphicalwindow in accordance with one or more modifications, the one or moremodifications comprising a first modification implemented when two ofthe plurality of elements are contacting opposite edges of the displayedgraphical window, the first modification changing at least the positionand/or shape of the displayed graphical window in accordance with theidentified positions and movement of the two elements contacting theopposite edges, a second modification implemented when two of theplurality of elements are contacting adjacent edges of the displayedgraphical window, the second modification changing at least the shapeand/or size of the displayed graphical window in accordance with theidentified positions and movement of the two elements contacting theadjacent edges, a third modification implemented when two of theplurality of elements are contacting a same edge of the displayedgraphical window, the third modification changing at least the positionand/or shape of the displayed graphical window in accordance with theidentified positions and movement of the two elements contacting thesame edge, the controller controlling the display device to display onthe display surface at least a second portion of the image within themanipulated graphical window, positions on the display surface of commonportions of the first and second portions of the image beingsubstantially the same.

For each of the above-summarized additional method and systemembodiments of the present invention, various aspects and features ofsuch embodiments may be carried out in accordance with the presentinvention. Some of these features and aspects are summarized below.

As an aspect of such embodiments of the present invention, the firstmodification comprises changing the shape of the displayed graphicalwindow by positioning a first of the opposite edges of the displayedgraphical window to align with the identified position of a first of thetwo elements contacting opposite edges, and by positioning a second ofthe opposite edges of the displayed graphical window to align with theidentified position of a second of the two elements contacting oppositeedges.

As another aspect of such embodiments, the first modification compriseschanging the vertical or horizontal position of the displayed graphicalwindow on the display device in accordance with a change in an axisdefined by the positions of the two elements contacting the oppositeedges.

As a further aspect of such embodiments, the second modificationcomprises uniformly scaling the size of the displayed graphical windowin accordance with the identified movement of the two elementscontacting the adjacent edges.

As yet another aspect of such embodiments, the third modificationcomprises changing the shape of the displayed graphical window bychanging a length of said same edge in accordance with a change indistance between the positions of the two elements contacting the sameedge.

In accordance with yet a further method embodiment of the presentinvention, a method of interfacing with a multi-input display devicecomprises displaying on a multi-input display device a graphical windowand a first portion of an image within the graphical window, thedisplayed graphical window having an inner frame and an outer frame,identifying an initial position of one or more elements contacting theinner frame or the outer frame of the displayed graphical window,identifying a movement of the one or more elements, changing at leastthe position, shape and/or size of the displayed graphical window as afunction of the initial position and movement of the one or moreelements, manipulating the image displayed within the displayedgraphical window in accordance with one or more modifications, the oneor more modifications comprising a first modification implemented whenthe initial position of the one or more elements is identified ascontacting the inner frame or the outer frame of the displayed graphicalwindow, the first modification comprising manipulating the imagedisplayed by displaying a second portion of the image within the changedgraphical window, and positions on the display device of common portionsof the first and second portions of the image being substantially thesame, and, a second modification implemented when the initial positionof the one or more elements is identified as contacting the other frameof the displayed graphical window, the second modification comprisingmanipulating the image displayed by changing the first portion of theimage in a same manner the displayed graphical window is changed anddisplaying the changed first portion of the image within the changedgraphical window.

In accordance with yet a further system embodiment of the presentinvention, a multi-input display system comprises a display device fordisplaying on a display surface a graphical window and a first portionof an image within the graphical window, the display device adapted todetect elements simultaneously contacting the display surface, thedisplayed graphical window having an inner frame and an outer frame, acontroller adapted to identify an initial position of one or moreelements contacting the inner frame or the outer frame of the displayedgraphical window, to identify a movement of the one or more elements, tochange at least the position, shape and/or size of the displayedgraphical window as a function of the identified initial position andmovement of the one or more elements, and to manipulate the imagedisplayed within the displayed graphical window in accordance with oneor more modifications, the one or more modifications comprising a firstmodification implemented when the initial position of the one or moreelements is identified as contacting the inner frame or the outer frameof the displayed graphical window, the first modification comprisingmanipulating the image displayed by controlling the display device todisplay a second portion of the image within the changed graphicalwindow, and positions on the display surface of common portions of thefirst and second portions of the image being substantially the same, asecond modification implemented when the initial position of the one ormore elements is identified as contacting the other frame of thedisplayed graphical window, the second modification comprisingmanipulating the image displayed by changing the first portion of theimage in a same manner the displayed graphical window is changed andcontrolling the display device to display the changed first portion ofthe image within the changed graphical window.

For each of the above-summarized further method and system embodimentsof the present invention, various aspects and features of suchembodiments may be carried out in accordance with the present invention.Some of these features and aspects are summarized below.

As an aspect of these embodiments, the first modification is implementedwhen the initial position of an initial one of plural elements isidentified as contacting the inner frame or the outer frame of thedisplayed graphical window, and the second modification is implementedwhen the initial position of the initial element is identified ascontacting the other one of the inner frame or the outer frame of thedisplayed graphical window.

As another aspect of these embodiments, the number of elementscontacting the inner frame or the outer frame of the displayed graphicalwindow is identified, and a characteristic of the displayed graphicalwindow is changed, the characteristic corresponding to a firstcharacteristic when only one element is identified as contacting theinner frame or the outer frame of the displayed graphical window, thefirst characteristic being a position of the displayed graphical windowon the display device, and the characteristic corresponding to a secondcharacteristic when a plurality of elements are identified as contactingthe inner frame or the outer frame of the displayed graphical window,the second characteristic being at least the shape and/or size of thedisplayed graphical window, the first and second characteristics beingdifferent.

As a further aspect of these embodiments, the number of elementscontacting the inner frame or the outer frame of the displayed graphicalwindow is identified, and the displayed graphical window is changed inaccordance with one or more graphical window modifications, the one ormore graphical window modifications comprising a first graphical windowmodification implemented when only two elements are identified ascontacting the inner frame or the outer frame of the displayed graphicalwindow, the first graphical window modification changing at least theshape and/or size of the displayed graphical window as a function of theidentified initial positions and movement of the two elements, and asecond graphical window modification implemented when three elements areidentified as contacting the inner frame or the outer frame of thedisplayed graphical window, the second graphical window modificationchanging at least the position, shape and/or size of the displayedgraphical window as a function of the identified initial positions andmovement of the three elements, the first and second graphical windowmodifications being different.

As another aspect of these embodiments, the number of elementscontacting the inner frame or the outer frame of the displayed graphicalwindow is identified, and the displayed graphical window is changed inaccordance with one or more graphical window modifications, the one ormore graphical window modifications comprising a first graphical windowmodification implemented when only one element is identified ascontacting the inner frame or the outer frame of the displayed graphicalwindow, the first graphical window modification moving an edge of thedisplayed graphical window contacted by the one element in a directionof identified movement of the one element normal to an axis of the edge,a second graphical window modification implemented when a plurality ofelements are identified as contacting the inner frame or the outer frameof the displayed graphical window along different edges of the displayedgraphical window, the second graphical window modification moving eachof the edges of the displayed graphical window contacted by a respectiveone of the elements in a direction of identified movement of therespective element normal to an axis of the respective edge.

As a further aspect of these embodiments, the displayed graphical windowis changed in accordance with one or more graphical windowmodifications, the one or more graphical window modifications comprisinga first graphical window modification implemented when two elements areidentified as contacting opposite edges of the displayed graphicalwindow, the first graphical window modification changing at least theposition and/or shape of the displayed graphical window in accordancewith the positions and movement of the two elements contacting theopposite edges, a second graphical window modification implemented whentwo elements are identified as contacting adjacent edges of thedisplayed graphical window, the second graphical window modificationchanging at least the shape and/or size of the displayed graphicalwindow in accordance with the positions and movement of the two elementscontacting the adjacent edges, a third graphical window modificationimplemented when two elements are identified as contacting a same edgeof the displayed graphical window, the third graphical windowmodification changing at least the position and/or shape of thedisplayed graphical window in accordance with the positions and movementof the two elements contacting the same edge.

In accordance with an additional method embodiment of the presentinvention, a method of interfacing with a multi-input display devicecomprises displaying on a multi-input display device at least first andsecond graphical windows, displaying a first portion of an image withinthe first graphical window, displaying a second portion of the imagewithin the second graphical window, the first portion and the secondportion representing at least portions of the same image, identifying anumber, position and movement of a first element or a first plurality ofelements contacting the displayed first graphical window, identifying anumber, position and movement of a second element or a second pluralityof elements contacting the displayed second graphical window,manipulating the displayed first graphical window as a function of theidentified number, position and movement of the first element or thefirst plurality of elements contacting the first graphical window,manipulating the displayed second graphical window as a function of theidentified number, position and movement of the second element or thesecond plurality of elements contacting the second graphical window,displaying on the display device, upon manipulating the first graphicalwindow, at least a third portion of the image within the manipulatedfirst graphical window, and positions on the display device of commonportions of the first and third portions of the image beingsubstantially the same, and displaying on the display device, uponmanipulating the second graphical window, at least a fourth portion ofthe image within the manipulated second graphical window, and positionson the display device of common portions of the second and fourthportions of the image being substantially the same.

In accordance with an additional system embodiment of the presentinvention, a multi-input display system comprises a display device fordisplaying on a display surface at least first and second graphicalwindows, for displaying a first portion of an image within the firstgraphical window, and for displaying a second portion of the imagewithin the second graphical window, the first portion and the secondportion representing at least portions of the same image, the displaydevice adapted to detect one or more elements contacting the displaysurface, a controller adapted to identify a number, position andmovement of a first element or a first plurality of elements contactingthe displayed first graphical window, to identify a number, position andmovement of a second element or a second plurality of elementscontacting the displayed second graphical window, to manipulate thedisplayed first graphical window as a function of the identified number,position and movement of the first element or the first plurality ofelements contacting the first graphical window, to manipulate thedisplayed second graphical window as a function of the identifiednumber, position and movement of the second element or the secondplurality of elements contacting the second graphical window, to controlthe display device to display on the display surface, upon manipulatingthe first graphical window, at least a third portion of the image withinthe manipulated first graphical window, positions on the display surfaceof common portions of the first and third portions of the image beingsubstantially the same, and to control the display device to display onthe display surface, upon manipulating the second graphical window, atleast a fourth portion of the image within the manipulated secondgraphical window, positions on the display surface of common portions ofthe second and fourth portions of the image being substantially thesame.

For the above-summarized additional method and system embodiments of thepresent invention, various aspects and features of such embodiments maybe carried out in accordance with the present invention. Some of thesefeatures and aspects are summarized below.

As an aspect of such embodiments, the displayed first and secondgraphical windows have an overlapping region, and a part of the imagedisplayed within the overlapping region represents at least a part ofthe first portion displayed within the first graphical window and alsorepresents at least a part of the second portion displayed within thesecond graphical window.

As another aspect of such embodiments, the overlapping region isdisplayed with content having a predefined characteristic andnon-overlapping region or regions of the first and second graphicalwindows are displayed with content not having the predefinedcharacteristic.

As a further aspect of such embodiments, the image represents a map, afirst portion of the map is displayed in the overlapping region at afirst resolution and a second portion or portions of the map aredisplayed in a non-overlapping region(s)s at a second resolution, thefirst and second resolutions being substantially different.

As an additional aspect of the invention, the image represents a map, afirst portion of the map having real time traffic data is displayedwithin the overlapping region, and a second portion or portions of themap not having real time traffic data are displayed within anon-overlapping region or regions of the first and second graphicalwindows.

As yet a further aspect of the invention, the image represents aphotograph, a first portion of the photograph having an imageenhancement characteristic is displayed within the overlapping region,and a second portion or portions of the map not having the imageenhancement characteristic are displayed within a non-overlapping regionor regions of the first and second graphical windows.

As yet another aspect of the invention, the image represents informationcontaining hidden data, a first portion of the image is displayed withinthe overlapping region and at least a portion of the hidden data isrevealed within the overlapping region, and a second portion or portionsof the image are displayed within a non-overlapping region or regions ofthe first and second graphical windows and the non-overlapping region orregions do not reveal any of the hidden data.

As yet a further aspect of the invention, the first and second graphicalwindows are displayed with an overlapping region and respectivenon-overlapping regions. A portion of the image with a firstcharacteristic is displayed within the overlapping region, a portion ofthe image with a second characteristic is display within thenon-overlapping region of the displayed first graphical window, and aportion of the image with a third characteristic is display within thenon-overlapping region of the displayed second graphical window. Thefirst, second and third characteristics are different from one another.

As another aspect of the invention, the displayed first and secondgraphical windows have an overlapping region and at least the firstgraphical window has a non-overlapping region, and a part of the imagedisplayed within the entire second graphical window is displayed inaccordance with a predefined characteristic, and a part of the imagedisplayed within the non-overlapping region of the first graphicalwindow is displayed not in accordance with the predefinedcharacteristic.

As a further aspect of the invention, at least the shape and/or size ofboth the first and second graphical windows are uniformly changed as afunction of the position and movement of elements contacting both thefirst and second graphical windows.

As an additional aspect of the invention, portions of the imagedisplayed within the manipulated first and second graphical windows aremanipulated in a same manner the first and second graphical windows arechanged.

Various other objects, advantages and features of the present inventionwill become readily apparent to those of ordinary skill in the art, andthe novel features will be particularly pointed out in the appendedclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The following detailed description, given by way of example and notintended to limit the present invention solely thereto, will best beappreciated in conjunction with the accompanying drawings, wherein likereference numerals denote like elements and parts, in which:

FIGS. 1A and 1B are schematic illustrations used to describeoperations/techniques for controlling a window frame using one finger inaccordance with a first embodiment of the present invention;

FIGS. 2A and 2B are schematic illustrations used to describe operationsfor controlling a window frame using two fingers in accordance with thefirst embodiment of the present invention;

FIGS. 3A to 3F are schematic illustrations used to describe operationsfor controlling a window frame using three fingers in accordance withthe first embodiment of the present invention;

FIGS. 4A and 4B are schematic illustrations used to describe operationsfor controlling a window frame in accordance with a second embodiment ofthe present invention;

FIGS. 5A to 5C are schematic illustrations used to describe operationsfor controlling a window frame with fingers contacting opposite edges ofthe window frame in accordance with a third embodiment of the presentinvention;

FIGS. 6A and 6B are schematic illustrations used to describe operationsfor controlling a window frame with fingers contacting adjacent edges ofthe window frame in accordance with the third embodiment of the presentinvention;

FIGS. 7A to 7C are schematic illustrations used to describe operationsfor controlling a window frame with fingers contacting the same edge ofthe window frame in accordance with the third embodiment of the presentinvention;

FIGS. 8A to 8C are schematic illustrations used to describe operationsfor controlling a window frame in accordance with a fourth embodiment ofthe present invention;

FIGS. 9A to 9C are schematic illustrations used to describe additionaloperations for controlling a window frame in accordance with the fourthembodiment of the present invention, in which the window frame andcontent therein are locked;

FIGS. 10A and 10B are schematic illustrations used to describeoperations for controlling a window frame in accordance with a variationof the fourth embodiment of the present invention;

FIGS. 11A to 11C are schematic illustrations used to describe operationsfor controlling a window frame in accordance with another variation ofthe fourth embodiment of the present invention;

FIGS. 12A and 12B are schematic illustrations used to describeoperations for controlling multiple, independent window frames inaccordance with the present invention;

FIGS. 13A to 13C are schematic illustrations used to describe operationsfor controlling multiple, common display window frames in accordancewith the present invention;

FIGS. 14A to 14C are schematic illustrations used to describe operationsfor controlling multiple, common display window frames in accordancewith further embodiments of the present invention;

FIGS. 15A to 15E are schematic illustrations used to describe operationsfor controlling multiple, common display window frames in accordancewith additional embodiments of the present invention; and

FIG. 16 is a block diagram of a system including a display device and acontroller for carrying out the various operations and techniques of thepresent invention.

DETAILED DESCRIPTION OF THE INVENTION

The present invention pertains to techniques for interfacing withmulti-input devices, including multi-input displays, multi-touchdisplays/screens, multi-input capable touch tablets, multi-input devicesthat receive inputs via one or more fingers of a user and/or via one ormore styluses or other mechanical, electro-mechanic, magnetic, etc.,devices, and any other device or apparatus capable of sensingsimultaneous multiple inputs. The multi-input displays/screens may be inthe form of a computer monitor, a television, a telephone display, apersonal digital assistant, and other such devices, whether portable orstationary. Moreover, multiple devices, such as two or more computermonitors, may operate together to display collectively an image orimages to a user or users and, thus, the multi-input displays/screensmay entail multiple devices/systems. As used herein, the terms“multi-input device” or “multi-input display device” (or, forconvenience herein, “display device” or “input device”) areinterchangeably used herein and shall refer to any of theabove-mentioned devices including multiple units thereof or combinationsof such devices. In addition, for convenience, the present invention isdescribed in connection with an input device or display device receivingone or more inputs in the form of a user's finger or fingers contactingthe input device. It is appreciated, however, that other forms of inputsmay be employed, such as via a stylus as mentioned above. Accordingly,the terms “finger” and “element” are interchangeably used herein torefer to any item (i.e., element) contacting the display device inaccordance with the present invention.

The present invention pertains, in particular, to a user interfacewindowing system. As further described below and with particularreference to the drawing figures, the present invention provides varioustechniques for controlling, including moving, sizing, orientating, amongother things, one or more windows in which an image or images aredisplayed. As discussed and shown in the various embodiments set forthherein, the window corresponds to a window frame through which (or inwhich) an image is provided and wherein parts of the image that extendbeyond the window frame are not displayed. As described in detail below,the present invention is a user interface for controlling the windowing(or window framing) of images.

Multi-Touch Enabled Rectilinear Window Frame Style 1

FIGS. 1A-1B, 2A-2B, and 3A-3C schematically describe a first embodimentfor controlling a window frame by employing both single and multi-touchtechniques as herein described. For convenience only, the firstembodiment is referred to herein as window frame style 1. In accordancewith the present invention, window frame style 1 operates in differentmanners depending on the number of contact points (e.g., number of userfingers employed), the relative positions of those contacts and therelative movement of those contact points.

FIG. 1A of the drawings shows a display device 10 that displays an imageof a window 12 (also called “window frame”) in which an image 14(sometimes referred to herein as a “picture”) is provided. Conceptually,picture 14 extends beyond window 12, but only that portion disposedwithin window 12 is displayed. Window frame style 1 includes controllingthe position of window 12 by moving a single contact point (e.g., usinga single finger f1). In particular, sliding one contact point fromposition A1 to a different position A2, as schematically shown in FIG.1A, moves window 12 from the position shown in FIG. 1A to the positionshown in FIG. 1B. However, as shown, the position of picture 14 remainsstationary relative to the display device 10 and thus the actual imagedisplayed when the window is at one position differs from the imagedisplayed when the window is at a different position.

Stated differently, a first portion of an image is displayed within thewindow when the window is located at one position (e.g., see FIG. 1A),and a second portion of that same image is displayed within the windowwhen the window is at a different position (e.g., see FIG. 1B). But theentire image (which usually is not displayed in its entirety) remainsstationary relative to the display device and, thus, positions on thedisplay device of the common portions of the above-mentioned first andsecond portions are the same.

The picture 14 shown in the exemplary figures is a well-recognized imageof the Earth and is shown in the figures to facilitate an understandingof the various embodiments and features of the present invention.However, any picture or image, including text, may be displayed withinthe window.

As mentioned above, and with reference again to FIGS. 1A and 1B, it isseen that moving window 12 does not move the displayed image 14 relativeto the display device 10. Accordingly, the present invention iswell-suited for mapping, editing and other useful graphical functions.

Window frame style 1 further includes controlling window 12 by using twopoints of contact (e.g., first and second fingers f1 and f2), asillustrated in FIGS. 2A and 2B. In particular, employing two points ofcontact moves and uniformly scales window 12. But, as with one fingercontrol, picture 14 remains stationary relative to display device 10during two-finger control. As shown in FIG. 2A, finger f1 slides fromcontact point B1 to contact point B2 while finger f2 slides from contactpoint C1 to contact point C2. Fingers f1 and f2 may move simultaneouslywith one another or in succession. As window 12 increases in size, theamount of image 14 that is displayed also increases, as shown in FIG.2B. Window rotation may also, optionally, be simultaneously controlledusing fingers f1 and f2 (e.g., see FIGS. 3D and 3E discussed below). Itis appreciated that while FIGS. 3D and 3E show two fingers disclosed ona single edge controlling window rotation, rotation may be implementedvia multiple fingers disposed at different positions, such as onadjacent edges. Hence, two input points (e.g., two fingers) may beutilized to effect the position, scaling and rotation, that is,pan-zoom-rotate (or “PZR”) of a window or, in other words, implementrotation, scale and translation (“RST”) control of window 12 using twopoints of contact. In certain applications, it may be desirable todisable rotation so that the window edges are always axis-aligned, whichmay be the case for any of the embodiments and variations thereofdescribed herein.

Three points of contact may control a window, in accordance with thepresent invention, by applying a “best effort” or “best fit” methodologyto the manipulation of the window frame. In particular, the coordinatesof a rectangular window frame are computed to best fit the dynamicpositions of three (or more) fingers. As shown in the examples discussedbelow, three input points (e.g., three fingers) may be utilized toeffect the position, non-uniform (i.e. anisotropic, axis-independent)scaling and rotation (i.e., a 5 degree of freedom (5 DOF) operation) ofa window on a display based on a change from the initial positions ofthe points or based incrementally on changes in the positions of thepoints. In a particular implementation, two fingers disposed on one edgeof the window, for example, the left edge, locks that edge to thepositions of those two fingers, with a third finger disposed on anopposite edge to control that edge. Subsequently moving one or morefingers in turn enables the window to move, scale or rotate accordingly.

FIGS. 3A-3C illustrate how the respective positions/lengths of window12's edges may be controlled or, in other words, how the width and/orthe height of window 12, along with the window's position, may becontrolled utilizing first, second and third fingers f1, f2 and f3.Finger f1 is disposed on a first window edge and fingers f2 and f3 aredisposed on an opposite window edge to trigger 3-point of contact windowcontrol. The width of window 12 increases (or decreases) (or, stateddifferently, the position of the window's right edge moves) as finger f1moves horizontally. In FIG. 3A, finger f1 moves from contact point D1 tocontact point D2, thereby causing the right edge of window 12 to extendto contact point D2, as shown in FIG. 3B. Moving horizontally fingers f2and f3, contacting the same window edge, similarly controls thehorizontal position of that edge. In a variation, placement of twofingers on the same edge locks that edge in position, so that only theopposite edge moves horizontally when the finger contacting that edge(e.g., finger f1) moves horizontally. In another variation, the edge onwhich a single finger is contacting is locked in position, with thehorizontal position of the opposite edge being controlled by the othertwo fingers.

Height control of the window and/or the vertical position of the window(i.e., the respective positions of the upper and lower edges) similarlyis achieved, along with the variations mentioned above, by employingthree contact points on the upper and lower edges of the window.

In a further variation, fingers f2 and f3, contacting the same windowedge as shown in FIG. 3A, control the window's height (i.e., controlsthe distance between the edges not being contacted) as a function of therelative change in distance between those fingers. For example, slidingfingers f2 and f3 from their initial respective contact points E1 and F1to respective contact points E2 and F2, as shown in FIG. 3A, causes theheight of window frame 12 to increase in proportion to the proportionalincrease in the distance between the fingers f2 and f3, such as shown inFIG. 3C.

In a variation, the horizontal position of the window's left edge shownin the figures (i.e., the edge with two points of contact) is locked sothat only window height is controlled by moving fingers f2 and f3. Inanother variation, the second and third fingers control both thehorizontal position of the edge being contacted and the relativedistance between the adjacent edges (i.e., the upper and lower edges).Two points of contact on either the upper or lower window edgessimilarly controls the vertical position of that edge and/or therelative distance between the window's two side edges.

In accordance with the present invention, the window may be rotated asschematically illustrated in FIGS. 3D and 3E. As shown in FIG. 3D,fingers f2 and f3 initially contact the same window edge thereby lockingthat edge of the window to the axis that extends through the contactpositions of fingers f2 and f3. As finger f2 and/or finger f3 moves, thewindow edge follows the axis that extends through the fingers' contactpoints, as shown in FIG. 3E.

FIGS. 3E and 3F collectively illustrate that the window's size may becontrolled after the window has been rotated. The window's size, as wellas its position, may also be controlled simultaneously while (or priorto) the window is rotated. For example, extending and rotating twofingers contacting the same window edge both rotates that window edgeand extends the window's size along that axis. Moving a third finger on,for example, the opposite window edge further simultaneously increasesthe window's size accordingly, as previously discussed. Multiple otherforms of simultaneous control of window position, size and angularposition (anisotropic RST control) also are possible by employing theherein-described control techniques.

The above-described techniques may be implemented, in accordance withthe present invention, by having two points of contact (e.g., fingers f2and f3) on one of the window's edges in order to control theposition/scale/rotation of the line segment that corresponds to (i.e.,represents) that edge. In other words, the two points that delineatethis line segment are directly controlled by two points of contact. Thethird point of contact (e.g., finger f1) then uniquely specifies a linethat is parallel to this line segment, with such specified linerepresenting the window's opposite edge. These two line segments, fullycontrolled by the three points of contact, then specify therectangular-shaped window.

Multi-Touch Enabled Rectilinear Window Frame Style 2

FIGS. 4A-4B schematically show a second embodiment for controlling awindow frame, wherein each contact point on an edge adjusts that edgealong a direction normal to it. For example, four fingers f1, f2, f3 andf4 are shown in FIG. 4A contacting four different edges of window 12.Moving any finger in a direction normal to the axis of that edge (e.g.,while holding another finger on another edge) moves that edge in thedirection of movement of the finger. In FIG. 4A, finger f1 slides fromcontact point G1 to contact point G2 (e.g., while finger f3 remains atposition I1), thus causing the window's right edge to move toward theright as shown in FIG. 4B. Similarly, moving finger f2 down from contactpoint H1 to contact point H2 (e.g., while finger f4 remains in positionJ1) causes the window's top edge to move down. Moving fingers f3 and f4likewise control the respective positions of the window's left andbottom edges.

In window frame style 2, the position of each and every edge of thewindow is controlled separately by moving a finger (or other contactitem) contacting that edge (e.g., relative to a finger contacting anopposite edge). A single edge may be controlled with a single finger.Two edges may be controlled simultaneously or in succession with twofingers. Likewise, three or four edges may be controlled simultaneouslyor in succession with three or four fingers, respectively.

Multi-Touch Enabled Rectilinear Window Frame Style 3

FIGS. 5A-5C, 6A-6B, and 7A-7C schematically show a third embodiment forcontrolling a window frame, referred to herein for convenience as windowframe style 3. In window frame style 3, the user may control the windowas herein described by utilizing one, two or three points of contact(e.g., one, two or three fingers). A single point of contact moves thewindow frame in the same manner as previously described above withreference to FIGS. 1A and 1B.

Two points of contact control the window frame in various differentmanners depending upon the particular edges each of the two fingerscontacts. FIGS. 5A-5C schematically illustrate window frame control whentwo points of contact (e.g., fingers f1 and f2) are disposed on oppositeedges of window 12. In this initial configuration, the two contactpoints adjust the window to maintain distance and position along theaxis that is defined by the two points of contact. More specifically,two points of contact (e.g., fingers f1 and f2) initially are disposedon contact points K1 and L1, as shown in FIG. 5A. Sliding finger f1 fromcontact point K1 to contact point K2 moves the axis on which finger f1is disposed horizontally to contact point K2, as shown in FIG. 5B. Inaddition, the position of the axis as defined by the two contact pointscontrols the vertical position of the window or, in other words, thevertical position of the axis defined by the two points of contact. Forexample, moving fingers f1 and f2 from contact points K2 and L1,respectively, as shown in FIG. 5B, to contact points K3 and L2 causeswindow 12 to follow such movement and to move down the display to theposition shown in FIG. 5C. Placing two contact points on the upper andlower edges of the window and subsequent movement of these contactpoints similarly controls the height and horizontal position of window12 within the display.

In accordance with the embodiment of window frame style 3, two points ofcontact disposed on adjacent edges of window 12 moves and uniformlyscales window 12, as shown in FIGS. 6A and 6B. As shown in FIG. 6A,fingers f1 and f2, initially disposed on contact points M1 and N1, slideto contact points M2 and N2, thereby causing a uniform increase in thewindow size as shown in FIG. 6B.

Two points of contact disposed on the same edge of window 12 controlsthe window in yet a different manner, namely the scale and position ofthat edge, as shown in FIGS. 7A, 7B and 7C. Fingers f1 and f2 initiallyare disposed on contact points O1 and P1, respectively, as shown in FIG.7A. Uniform movement downward of the contact points to points O2 and P2causes the relative position of the window to follow such movement, asshown in FIG. 7B. The two contact points positioned on the same edgefurther control the relative scale, that is, length of the edge.Specifically, increasing the distance between the two fingers, forexample, sliding the fingers from respective contact points O2 and P2 tocontact points O3 and P3, as shown in FIG. 7B, results in a proportionalincrease in the height of window 12, as shown in FIG. 7C.

Multi-Touch Enabled Rectilinear Window Frame Style 4

FIGS. 8A-8C and 9A-9C schematically show a fourth embodiment forcontrolling a window frame, referred to herein for convenience as windowframe style 4. As shown in each of these figures, a window frame 20 is asort-of double edge window and includes an inner frame 22 (or inneredge) and an outer frame 24 (or outer edge) in which a picture (or otherimage) 26 is disposed. Window frame style 4 operates in differentmanners depending on the number of contact points, the relativepositions of those contacts including whether the contact points aredisposed on the window's inner frame or its outer frame, and themovement of those contact points.

In accordance with the present invention, contacting the window's innerframe 22, such as shown in FIGS. 8A-8C, operates to control the windowin any of the styles discussed above. For example, similar to theoperation of window frame style 1 described above and described withreference to FIGS. 3A-3C, FIGS. 8A-8C illustrate how the width and/orthe height of window 20 may be controlled utilizing three points ofcontact (e.g., fingers f1, f2 and f3). As shown in FIG. 8A, finger f1initially is disposed on one edge of inner frame 22, and fingers f2 andf3 are then disposed on the opposite edge of inner frame 22 in order totrigger 3-point of contact control of the window. The width of window 20increases (or decreases) with the horizontal movement of finger f1. Asparticularly shown in FIG. 8A, finger f1 moves from contact point Q1 tocontact point Q2, thereby causing the right edge of window 20 to extendto contact point Q2, such as shown in FIG. 8B. Moving fingers f2 and f3horizontally similarly controls the horizontal position of that edge.Like the embodiment of window style 1, a variation encompasses placingtwo fingers on the same edge to lock that edge in position, so that onlythe opposite edge moves horizontally when the finger contacting thatedge of the inner frame 22 (e.g., finger f1) moves horizontally. Inanother variation, the edge on which a single finger is contacting islocked in position, with the horizontal position of the opposite edgebeing controlled by the other two fingers.

Height control of the window similarly is achieved, along with thevariations mentioned above, by employing three contact points on theinner frame 22 of the upper and lower edges of the window 20.

In a further variation, spreading of fingers f2 and f3 apart (or movingtogether) from contact points R1 and S1, respectively, as shown in FIG.8A, to contact points R2 and S2, as shown in FIG. 8C, controls thewindow's height to increase (or decrease) in proportion to theproportional increase (or decrease) in the distance between the fingersf2 and f3. Other variations mentioned above also may be employed.

In each of the embodiments and variations mentioned above, moving thewindow frame and/or increasing the size of the window frame (verticallyand/or horizontally) and/or decreasing the size of the window frame(vertically and/or horizontally) is carried out by the use of one, two,three or four points of contact, where such window movement and/orwindow change does not cause any movement or change in size of thepicture or image displayed within the frame relative to the displaydevice.

In accordance with the present invention, window frame style 4 furtherprovides user movement and/or size (and rotation) control of the imagedisplayed within the window by employing, that is, contacting, thewindow's outer frame 24. In particular, in each of the embodiments andvariations described herein, initial placement of the contact point orcontact points on the window's outer frame 24 locks the window frame andthe content displayed therein together so that both window and contentposition and size are together controlled by the subsequent movement ofone or more contact points. For example, FIGS. 9A-9C schematicallyillustrate controlling window 20 and content 26 therein by initiallycontacting (e.g., with fingers f1, f2 and f3) the outer frame 24 ofwindow 20. As shown in FIG. 9A, finger f1 contacts one edge of outerframe 24, and fingers f2 and f3 contact the opposite edge of outer frame24, thus locking the window frame and content therein together. Movingfinger f1 from contact point T1 to contact point T2 controls both window20 and image 26 displayed therein to stretch, thus producing image 26′displayed within the enlarged window, as schematically shown in FIG. 9B.

Furthermore, spreading of fingers f2 and f3 apart (or moving together)from contact points U1 and V1, respectively, as shown in FIG. 9A, tocontact points U2 and V2, as shown in FIG. 9C, stretches (or shrinks)together both the window frame and content therein in proportion to theproportional increase (or decrease) in the distance between the fingersf2 and f3. As shown in FIG. 9C, image 26″ corresponds to image 26 butenlarged, or stretched, together with the enlarged window frame 20.

Each of the other forms of control, whether by one, two, three or fourpoints of contact, as described herein, may be employed in mannersconsistent with the operation of window frame style 4, includingrotational control of a window as previously discussed. Still yet,further variations may be employed. In one variation, the respectiveoperations achieved when the inner and outer frames are contacted arereversed, whereby window control alone (i.e., the image displayed withinthe window is unaffected) is achieved by contacting the outer frame, andcombined control of window and the image therein is achieved bycontacting the inner frame.

In yet another variation, the position of the first point of contactestablishes the type of window/image control to be carried out. That is,if the first contact point is on the window's inner frame, only windowcontrol is carried out regardless of whether the outer or inner frame issubsequently contacted by additional contact elements. Or, if the firstcontact point is on the window's outer frame, then the window andcontent therein are locked together regardless of whether the outer orinner frame is subsequently contacted by additional contact elements.For example, if fingers f1 and f2 initially contact outer window frame24, and finger f3 subsequently contacts inner window frame 22, as shownin FIG. 10A, then window and content control is actuated. Moving fingerf3 thus causes both window 20 and image 26 displayed therein to stretch,as shown in FIG. 10B.

In yet a further variation, the moving element establishes the type ofcontrol to be carried out. For example, FIG. 11A shows finger f1contacting inner window frame 22 and fingers f1 and f2 contacting outerwindow frame 24. The particular order in which the fingers contacted thewindow does not impact the type of window control to be carried out.Instead, moving finger f1 controls the size of window 20, withoutchanging the size of image 26, as shown in FIG. 11B since finger f1initially contacted inner frame 22. Conversely, moving finger f2 and/orfinger f3 controls the size of both window 20 and image 26, as shown inFIG. 11C, since fingers f2 and f3 initially contacted outer frame 24.

In the embodiments and variations described herein in which content sizeis controllable, such as when the window frame and the content displayedtherein are “locked” together, a preferable feature is to uniformlyscale the content along both axes, which is also known as “fixed aspectratio scaling.” In the various embodiments/features described herein,uniform scaling may be implemented by uniformly enlarging/shrinking thecontent proportional to the increase/decrease in size of only either thewidth or height of the frame in which the content is displayed. Otherknown methods to carry out such uniform scaling also may be implementedwithin the various embodiments described herein.

Multiple Windows

In accordance with the present invention, each of the window framestyles and variations thereof as described above may be applied to thedisplay and control thereof of multiple windows, referred to herein forconvenience as Multi-Window. Different manners of such display andcontrol may be applied as described below.

Multi-Window Style A—Independent Windows

Multi-Window Style A, in accordance with the present invention, displaysand controls each window in manners that are independent from thedisplay and control of other windows and provides for assigning priorityto any one of multiple overlapping windows. FIGS. 12A and 12B showwindows W1 and W2 displayed simultaneously, each window providingseparate and distinct images I1 and I2, and each window allowing forseparate control by any of the window frame styles described herein. Forexample, FIGS. 12A and 12B illustrate how the width and/or height ofwindows W1 and W2 may each be independently controlled using threepoints of contact under window frame style 1. Specifically, FIG. 12Ashows the horizontal width of window W1 being increased using threepoints of contact by moving finger f11 towards window W2. Similarly,FIG. 12A shows the horizontal width of window W2 also being increased,independent of window W1, by moving finger 21 towards window W1. WhileFIG. 12A specifically shows both windows being controlled using threepoints contact, any window frame method discussed herein may have beemployed. FIG. 12B shows Windows W1 and W2 in their enlarged states. Inthis frame style, since each window may display a different image(including text), one window generally is given visual priority, withany overlapping portion of the other window being visually blocked.Changing window priority may be achieved in any known manner.

Multi-Window Style B—Common Display Windows

Multi-Window Style B, in accordance with the present invention, controlseach window providing an image in manners that are independent from thecontrol of other windows that provide that same image. FIGS. 13A-13Cshow windows W1 and W2 displayed simultaneously, with each windowshowing a separate view (or portion) of the same image I1. Frames b1 andb2 of Windows W1 and W2, respectively, may be controlled separately aspreviously discussed. For example, the width and/or height of a windowmay be controlled using window frame style 1 previously discussed. Asshown in FIG. 13B, the windows are resized and/or moved so that theyoverlap with one another, with section O1 shown in FIG. 13B representingthe overlapping portion of the two windows. Since the two windowsdisplay portions of the same image I1, window visibility priority neednot be established.

Overlapping section O1 includes borders as shown in FIG. 13B. However,the borders of section O1 may be modified from that shown and, forexample, may be made thinner, hashed, lighter in color, different incolor, or different in another manner from the other border sections ofWindows W1 and W2. In one variation, the borders of section O1 may beremoved altogether, thereby providing an unobstructed view of theoverlapping section as shown in FIG. 13C. Moreover, while the variousviews show rectangular shaped windows, with at least FIG. 13B showing arectangular shaped overlapping section, the shape and size of eachwindow, as well as overlapping sections thereof, may have differentshapes, including round, oval, octagonal, heart-shaped, etc.

Multi-Window Style C—Lens Windows

Multi-Window Style C, in accordance with the present invention, includesmultiple windows that operate to display images in different manners.FIG. 14A shows two non-overlapping windows that may operate in mannerspreviously described. When the windows overlap, such as shown in FIG.14B, the image within the overlapping portion O1 is displayed in amanner that differs from the way the image is displayed in thenon-overlapping window portions.

In the embodiment of Multi-Window Style C, the overlapping portion O1operates as a sort of lens that displays a characteristic (orenhancement) of that portion of the image within overlapping portion O1that is not visually provided elsewhere within the windows (i.e., withinthe non-overlapping portions). As one example, when the displayed imageis a map such as shown in the figures, the image is displayed at agreater resolution within overlapping portion O1 than is displayed innon-overlapping portions of each window. In FIG. 14B, the image withinportion O1 is shown lightened, as compared to other displayed portionsof the map to illustrate such a different characteristic or enhancement.As a further example, the characteristic displayed within overlappingportion O1 is real time traffic data wherein the image that is displayedis a map that include street and other information useful to a traveler.As yet another example, the characteristic displayed may be a preview ofthe outcome of image processing, such as red eye reduction, monochromeconversion, or magnification. As yet a further example, in the case theinformation displayed within the windows is a computer-type file thatincludes embedded or hidden data (e.g., meta-data), such information maybe made available for viewing and/or amending within the overlappingportion O1. Since the size and/or shape of the lens, that is, theoverlapping portion O1 is user modifiable, the amount of additional ordifferent information that is made available within the “lens” may bemade as large or small as needed or desired.

FIG. 14C shows yet a further variation of Multi-Window Style C, whereinsection L1 is defined by the overlapping window portions. In thisvariation, each of at least two windows displays a respective portion ofthe same image in a different manner. In particular, window W1 in FIG.14C displays a first characteristic of a portion of an image such as,for example, the temperature of the various regions displayed withinwindow W1. Window W2, at the same time, displays a second characteristicof the image such as, for example, the relative altitude of thedisplayed image. In accordance with this variation, the overlappingportion L1 displays yet a third characteristic of that portion of thedisplayed image such as, for example, an amount of rain-fall via acolor-coded key. In a variation, the overlapping portion L1 displaysboth the first and second characteristics of the image.

Multi-Window Style D—Lens Windows II

Multi-Window Style D, in accordance with the present invention, includesmultiple windows that display the same image, similar to styles B and C,but where the windows have layering, such as shown in style A. Morespecifically, and with reference to FIG. 15A, window W1 displays animage while window W2, which has visibility priority over window W1,operates as the lens that displays a characteristic or enhancement ofthe displayed image that is otherwise not provided within window W1. Thecharacteristic or enhancement shown within window W2 may be an imageprocessed version of the image that is displayed in window W1, or othercharacteristic/enhancement mentioned herein, or any characteristic orenhancement. Rotation, scale, and translation (RST) control of windowsW1 and W2 may be carried out in any suitable manner, including thosemethods described herein.

For illustrative purposes, FIG. 15B shows window W2 fully within windowW1, which may be the case for any of the multi-window embodimentsdescribed herein. As mentioned above, windows W1 and W2 may becontrolled in various manners. For example, FIG. 15C shows window W2moved to a different position (still within window W1), such as, forexample, by employing single finger control as described in connectionwith style 1 (see FIGS. 1A and 1B). As shown in FIGS. 15B and 15C,window W2 may be moved, and well as its size and orientation modified,without disturbing window W1.

As mentioned above, each of the multiple windows displayed within thedisplay device may be controlled in accordance with any of theembodiments and variations described herein. In certain embodiments,rotation/scale/translation (RST) control of the image/content displayedwithin the window may be employed, such as discussed in connection with,for example, Style 4 (e.g., see FIGS. 9A-9C). When the window frame andthe content therein are “locked” together, both window and contentposition, size, and rotation are controlled together. In accordance witha particular feature of the present invention, a change of scale of onewindow frame when locked to the content therein (preferably, uniformscaling as mentioned above) causes a change of scale of all windowframes displaying that content as a group. For example, FIGS. 15D and15E illustrate a uniform scaling operation that is carried out whenwindow W1 and the content therein are locked together. In the exampleshown in FIG. 15D, finger f1 contacting the outer edge of window W1locks window W1 and the content therein together, and finger f2, bysubsequently contacting an edge of window W2, is able to carry out RSTcontrol of the windows and content therein to produce the result shownin FIG. 15E. Hence, as the image/content increases (or decreases) insize, so do the multiple windows that display that content. Othermanners of locking together the window and content, and controlling thescale thereof may also be employed. This feature of the presentinvention may be applied to any of the multi-window control stylesdescribed herein wherein multiple windows show different views of thesame image/content, including multi-window styles B and C describedabove. Still further, the uniform scaling operation of the windows shownin FIGS. 15D and 15E may be employed, but without modifying the contentwithin the windows, such as by contacting the inner edge of window W1and subsequently contacting an edge of window W2. Then, subsequentmovement of the fingers carries out RST control of the windows, but notof the content therein. Also, such as mentioned above, other manners oflocking the windows together (but not content therein) may be employed.

As herein described, the multi-window styles enable for rotation, scaleand translation (RST) control of multiple windows displayedsimultaneously to a user, with each window showing either differentimages or different portions of the same image. In certain embodiments,different windows show the same characteristic or characteristics orfeatures of the image while other characteristics or enhancedcharacteristics are provided within those portions of the windows thatoverlap with one another. In yet further embodiments and variationsthereof, different windows each show different characteristics orfeatures of the image, with their overlapping portions still yet showingadditional and/or different information to the user. In the variousfigures presented, two rectangular-shaped windows are provided for easeof illustration, but more than two windows may be employed and/ordisplayed simultaneously, and the windows may have the same or differentshapes as previously mentioned. In certain circumstances, multipleoverlapping portions will be defined, generally depending on the size,shape and position of each window employed. The multiple overlappingwindows may operate in the same manner, with each overlapping portiondisplaying the same characteristic of the image as the other overlappingportions. In a variation, the different overlapping portions showdifferent characteristics or employ different functionality. In yet afurther variation, three or more windows may overlap at the samelocation thus forming an overlapping portion that may yet show adifferent characteristic or feature, and/or operate in a mannerdifferent from that of overlapping portions formed from only twowindows.

As described herein, multiple embodiments of the present invention havebeen described in connection with the use of one, two, three, etc.,fingers (contact points) contacting various parts of a window or windowsdescribed herein. In particular, various embodiments entail one or morefingers contacting one or more edges of a window that, in turn, enablesparticular control of the window depending on various factors, such asparticular placement of the fingers (e.g., whether on the same edge,adjacent edges, etc.), subsequent movement of the fingers, and so on.The present invention, however, is not limited to placement of aparticular finger solely on a single edge, but also encompassesmanipulating a window by allowing a finger to be placed on a corner of awindow, wherein such finger is deemed to be contacting two adjacentedges. Then, the window is manipulated in accordance with themanipulation rules that have been described. As one example, style 2 asherein described (with reference to FIGS. 4A and 4B) may be modified toallow a finger (e.g., finger f1) to contact a corner of the window(e.g., the lower, right hand corner) thus allowing the user to controltwo edges (e.g., the lower edge and the right-hand edge) via subsequentmovement of that finger. In the exemplary diagram shown in FIG. 4A, allfour edges are controlled with only 3 fingers (e.g., fingers f1, f2 andf3), without the need to employ a fourth finger (e.g., finger f4).

In addition to enabling a single finger to be a control for multiple,adjacent edges by contacting a corner of a window, in certainembodiments multiple fingers can be a control for a single edge. Forexample, FIGS. 4A and 4B show each edge being controlled by a singlefinger, but in a variation of style 2 described herein, there may bemultiple fingers controlling a given edge (e.g., two fingers controllingthe top edge).

Various embodiments for interfacing with multi-input devices, andvarious features and variations thereof, have been described. Thepresent invention also encompasses a system designed to carry out thevarious interfacing techniques and processes described herein. Forexample, FIG. 16 is a block diagram of a multi-input display system 60that includes a display device 62 coupled (directly or indirectly) to acontroller 66. Display device 62 includes a display surface 64 (alsocalled contact surface) and may be any suitable type of multi-inputdisplay device capable of detecting multiple inputs simultaneously.Various suitable multi-input display devices that may be employedinclude those described in U.S. Patent Publication No. 2008/0029691A1,which is incorporated herein by reference. Controller 66 operates tocarry out the various processing functions described herein, and may bea pre-programmed general purpose computer or other known system ordevice that carries out the novel functions/steps as previouslydescribed. Controllers suitable for use within the present invention arewell known, and it is within the abilities of one of ordinary skill inthe art to design and/or program a controller to implement theprocesses, techniques and features of the present invention, given thedescription provided herein. By way of example only, the system of thepresent invention may be embodied within a programmed computer systemthat is coupled to or has an integrated multi-touch display device. Byway of another example, the system of the present invention may beembodied within a portable multifunctional device (e.g., PDA) thatcontains an integrated multi-touch display. Various examples of portablemultifunctional devices that may be modified or otherwisedesigned/programmed to implement the processes/techniques of the presentinvention are disclosed in U.S. Pat. No. 7,479,949, which isincorporated herein by reference. Accordingly, the present inventionencompasses a system that includes a display device and a controllercapable of implementing the above-described techniques and processes.Consistent with other variations described herein, display device 62 mayinclude a multi-input device and, as a separate element, a displaydevice.

In describing the present invention as set forth herein, the terms“first”, “second”, and “third”, etc., are used to distinguish oneelement, thing, contact, etc., from another, and are not used todesignate relative position or arrangement in time, unless otherwisestated explicitly. For example, the phrases “first modification” and“second modification” as used herein does not mean that the secondmodification follows in time the first modification, but rather that the“first modification” and the “second modification” are distinct from oneanother.

The present invention has been described in the context of a number ofembodiments, and for various ones of those embodiments, a number ofvariations and examples thereof. It is to be understood, however, thatother expedients known to those skilled in the art or disclosed hereinmay be employed without departing from the spirit of the invention.

Therefore, it is intended that the appended claims be interpreted asincluding the embodiments described herein, the alternatives mentionedabove, and all equivalents thereto.

1. A method of interfacing with a multi-input display device, comprisingthe steps of: displaying on a multi-input display device a graphicalwindow and at least a first portion of an image within the graphicalwindow; identifying a position and movement of one element contacting anedge of the displayed graphical window or a plurality of elementssimultaneously contacting respectively different edges of the displayedgraphical window; manipulating the displayed graphical window inaccordance with one or more modifications, the one or more modificationscomprising: a first modification implemented when only one element iscontacting an edge of the displayed graphical window, the firstmodification moving the edge of the displayed graphical window contactedby the element in a direction of identified movement of the one elementnormal to an axis of the edge; a second modification implemented when aplurality of elements are simultaneously contacting respectivelydifferent edges of the displayed graphical window, the secondmodification simultaneously moving each of the edges of the displayedgraphical window contacted by a respective one of the elements in adirection of identified movement of the respective element normal to anaxis of the respective edge; displaying on the display device at least asecond portion of the image within the manipulated graphical window, andpositions on the display device of common portions of the first andsecond portions of the image being substantially the same.
 2. The methodof claim 1, wherein identifying a position and movement comprisesidentifying a position and movement of three or more elementssimultaneously contacting three or more different edges of the displayedgraphical window; and the second modification simultaneously moves eachof the three or more edges of the displayed graphical window contactedby the three or more elements in a respective direction of identifiedmovement of the respective element normal to an axis of the respectiveedge.
 3. A multi-input display system, comprising: a display device fordisplaying on a display surface a graphical window and at least a firstportion of an image within the graphical window, the display deviceadapted to detect one or more elements contacting the display surface; acontroller for identifying a position and movement of one elementcontacting an edge of the displayed graphical window or a plurality ofelements simultaneously contacting respectively different edges of thedisplayed graphical window, the controller adapted to manipulate thedisplayed graphical window in accordance with one or more modifications,the one or more modifications comprising: a first modificationimplemented when only one element is contacting an edge of the graphicalwindow, the first modification moving the edge of the graphical windowcontacted by the element in a direction of identified movement of theone element normal to an axis of the edge; a second modificationimplemented when a plurality of elements are simultaneously contactingrespectively different edges of the graphical window, the secondmodification simultaneously moving each of the edges of the graphicalwindow contacted by a respective one of the elements in a direction ofidentified movement of the respective element normal to an axis of therespective edge; the controller controlling the display device todisplay on the display surface at least a second portion of the imagewithin the manipulated graphical window, positions on the displaysurface of common portions of the first and second portions of the imagebeing substantially the same.
 4. The system of claim 3, wherein thecontroller is adapted to identify a position and movement of three ormore elements simultaneously contacting three or more different edges ofthe displayed graphical window; and the second modificationsimultaneously moves each of the three or more edges of the displayedgraphical window contacted by the three or more elements in a respectivedirection of identified movement of the respective element normal to anaxis of the respective edge.
 5. A method of interfacing with amulti-input display device, comprising the steps of: displaying on amulti-input display device a graphical window and at least a firstportion of an image within the graphical window; identifying positionsand movement of a plurality of elements simultaneously contacting one ormore edges of the displayed graphical window; manipulating the displayedgraphical window in accordance with one or more modifications, the oneor more modifications comprising: a first modification implemented whentwo of the plurality of elements are contacting opposite edges of thedisplayed graphical window, the first modification changing at least oneof a position and shape of the displayed graphical window in accordancewith the identified positions and movement of the two elementscontacting the opposite edges; a second modification implemented whentwo of the plurality of elements are contacting adjacent edges of thedisplayed graphical window, the second modification changing at leastone of a shape and size of the displayed graphical window in accordancewith the identified positions and movement of the two elementscontacting the adjacent edges; a third modification implemented when twoof the plurality of elements are contacting a same edge of the displayedgraphical window, the third modification changing at least one of aposition and shape of the displayed graphical window in accordance withthe identified positions and movement of the two elements contacting thesame edge; displaying on the display device at least a second portion ofthe image within the manipulated graphical window, and positions on thedisplay device of common portions of the first and second portions ofthe image being substantially the same.
 6. The method of claim 5,wherein the first modification comprises changing the shape of thedisplayed graphical window by positioning a first of the opposite edgesof the displayed graphical window to align with the identified positionof a first of the two elements contacting opposite edges, and bypositioning a second of the opposite edges of the displayed graphicalwindow to align with the identified position of a second of the twoelements contacting opposite edges.
 7. The method of claim 5, whereinthe first modification comprises changing one of a vertical andhorizontal position of the displayed graphical window on the displaydevice in accordance with a change in an axis defined by the positionsof the two elements contacting the opposite edges.
 8. The method ofclaim 5, wherein the second modification comprises uniformly scaling thesize of the displayed graphical window in accordance with the identifiedmovement of the two elements contacting the adjacent edges.
 9. Themethod of claim 5, wherein the third modification comprises changing theshape of the displayed graphical window by changing a length of saidsame edge in accordance with a change in distance between the positionsof the two elements contacting said same edge.
 10. A multi-input displaysystem, comprising: a display device for displaying on a display surfacea graphical window and at least a first portion of an image within thegraphical window, the display device adapted to detect elementssimultaneously contacting the display surface; a controller foridentifying positions and movement of a plurality of elementssimultaneously contacting one or more edges of the displayed graphicalwindow, and for manipulating the displayed graphical window inaccordance with one or more modifications, the one or more modificationscomprising: a first modification implemented when two of the pluralityof elements are contacting opposite edges of the displayed graphicalwindow, the first modification changing at least one of a position andshape of the displayed graphical window in accordance with theidentified positions and movement of the two elements contacting theopposite edges; a second modification implemented when two of theplurality of elements are contacting adjacent edges of the displayedgraphical window, the second modification changing at least one of ashape and size of the displayed graphical window in accordance with theidentified positions and movement of the two elements contacting theadjacent edges; a third modification implemented when two of theplurality of elements are contacting a same edge of the displayedgraphical window, the third modification changing at least one of aposition and shape of the displayed graphical window in accordance withthe identified positions and movement of the two elements contacting thesame edge; the controller controlling the display device to display onthe display surface at least a second portion of the image within themanipulated graphical window, positions on the display surface of commonportions of the first and second portions of the image beingsubstantially the same.
 11. The system of claim 10, wherein the firstmodification comprises changing the shape of the displayed graphicalwindow by positioning a first of the opposite edges of the displayedgraphical window to align with the identified position of a first of thetwo elements contacting opposite edges, and by positioning a second ofthe opposite edges of the displayed graphical window to align with theidentified position of a second of the two elements contacting oppositeedges.
 12. The system of claim 10, wherein the first modificationcomprises changing one of a vertical and horizontal position of thedisplayed graphical window in accordance with a change in an axisdefined by the positions of the two elements contacting the oppositeedges.
 13. The system of claim 10, wherein the second modificationcomprises uniformly scaling the size of the displayed graphical windowin accordance with the identified movement of the two elementscontacting the adjacent edges.
 14. The system of claim 10, wherein thethird modification comprises changing the shape of the displayedgraphical window by changing a length of said same edge in accordancewith a change in distance between the positions of the two elementscontacting said same edge.